Spatio-temporal interaction model for crowd video analysis

We present an unsupervised approach to analyze crowd at various levels of granularity − individual, group and collective. We also propose a motion model to represent the collective motion of the crowd. The model captures the spatio-temporal interaction pattern of the crowd from the trajectory data captured over a time period. Furthermore, we also propose an effective group detection algorithm that utilizes the eigenvectors of the interaction matrix of the model. We also show that the eigenvalues of the interaction matrix characterize various group activities such as being stationary, walking, splitting and approaching. The algorithm is also extended trivially to recognize individual activity. Finally, we discover the overall crowd behavior by classifying a crowd video in one of the eight categories. Since the crowd behavior is determined by its constituent groups, we demonstrate the usefulness of group level features during classification. Extensive experimentation on various datasets demonstrates a superlative performance of our algorithms over the state-of-the-art methods. Understanding human behavior at an individual level, at a group level and at a crowd level in different scenarios has always attracted the researchers. The variability and complexity in the behavior make it a highly challenging task. However, this decade is witnessing a huge interest of researchers in the area of crowd motion analysis due to its various applications in surveillance, safety, public place management, hazard prevention, and virtual environments. This interest has resulted in many interesting papers in the area. We are aware of at least four survey papers on the subject of crowd analysis that indicate the amount of attention, it has drawn in this and the previous decade [1],[2],[3],[4]. The latest survey paper [1] by Chang et al. encapsulates the recent works published after 2009, covering topics of motion pattern segmentation, crowd behavior and anomaly detection. Thida et al. [2] provide a review on macroscopic and microscopic modeling methods. They also present a critical survey on crowd event detection. Julio et al. cover various vision techniques applicable to crowd analysis such as tracking, density estimation, and computer simulation [3]. Zhan et al. discuss various vision based techniques used in crowd analysis. They also discuss crowd analysis from the perspective of different disciplines − psychology, sociology and computer graphics [4]. At the top level, the techniques used in crowd motion analysis can be divided into two major classes − holistic and particle based. The holistic methods consider crowd as a single entity and analyze the overall behavior. These methods fail to provide much insight at an individual or intermediate level. On the other hand, particle based methods consider crowd as a collection of individuals. But their performance degrades with the increase in crowd density due to occlusion and tracking problems. The analysis at intermediate level i.e. at group level might provide more insights at individual and overall levels. We believe that a moderately dense crowd consists of various groups which form a primary entity of a crowd [6, 7] whereas a highly dense crowd can be considered to form a single group and a highly sparse crowd might have groups with cardinality of one. Together, they guide the overall behavior of the crowd and individually influence the actions of the members. Therefore, group level analysis and hence group detection becomes important in crowd analysis. We define a group as a set of individuals (agents) having some sort of interactions e.g. the group members are walking together. Spatial proximity is also necessary to form a group; if there are agents with a similar motion pattern but are far away from each other, they do not form a group as per our definition. Each group has its own set of goals that leads to various interaction patterns among the members of the group. The collective behavior of these constituent groups identifies the global crowd behavior which can vary from a highly structured to a completely unstructured pat1 ar X iv :1 71 0. 11 35 4v 1 [ cs .C V ] 3 1 O ct 2 01 7 (a) Uniform crowd (b) Mixed crowd (c) Stationary group (d) Walking (e) Approaching (f) Splitting Figure 1: (a) and (b) give examples of structured and unstructured crowd. Output of the proposed algorithm: (c) (f) show groups with different activities: Standing (St), Walking (W), Splitting (Sp) and Approaching (A). Tracklets for some of the agents over past few frames are also shown. Each color represents a detected group (Best viewed in color). The videos are from BEHAVE [5] and CUHK [6] datasets. tern. In case of a structured crowd, for example − marching of soldiers, all groups are in coordination and share the same goal (see Fig.1a); whereas in an unstructured crowd, for example − at railway station or at a shopping complex, there are multiple groups with different goals (see Fig.1b). We are interested in understanding these different types of crowd behaviors at various levels by exploiting motion information of individuals. The paper makes the following contributions: 1. A framework is proposed to model the collective motion of the crowd by a first order dynamical system. The model captures the interaction patterns among the individuals. Although, the proposed model does not capture any possible nonlinear relations, its usefulness for short-term analysis has been verified experimentally. We also provide an optimization formulation for the estimation of the interaction matrix under the constraints of spatial proximity, temporal continuity and sparsity of inter-agent relationship. 2. Since the interaction matrix is learned from the trajectory data, it captures the spatio-temporal patterns present among the agents. We observe that the eigenvectors of the interaction matrix reflect the spatio-temporal patterns. Thus, we propose a spectral clustering [8] based algorithm to identify the groups present in the scene. Extensive experimentation on various datasets demonstrates the effectiveness of the algorithm. 3. We also demonstrate how the activities can be classified at three different levels − at atomic (individual) level, at group level and at crowd level. The eigenvalues of the interaction matrix characterize various group and individual activities − Fig 1c-1f show examples of activities at group level. At crowd level, we employ group level features to identify the behavior of the crowd. We classify the crowd videos in one of the 8 categories as defined by [6] and demonstrate its performance in terms of classification accuracy. The remaining part of the paper is organized as follows. Next section reviews the related literature. Section 2 explains the proposed mathematical formulation followed by group detection algorithm in Section 3. Detection of group activity and atomic activity is discussed in Section 4. We look at crowd video classification in Section 5. The experimental results are presented in Section 6 followed by conclusions in Section 7.